Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task. We propose a search method for neural network architectures that can already perform a task without any explicit weight training. To evaluate these networks, we populate the connections with a single shared weight parameter sampled from a uniform random distribution, and measure the expected performance. We demonstrate that our method can find minimal neural network architectures that can perform several reinforcement learning tasks without weight training. On a supervised learning domain, we find network architectures that achieve much higher than chance accuracy on MNIST using random weights.

Originality: This paper draws from many fields (especially neural architecture search), but its core is a unique and powerfuly original idea. Quality: The execution of this idea by the authors is thorough and the results are compelling. The scholarship evident in this work is also exemplary. Clarity: This paper is extremely well-written and easy to follow. One small exception, however, was how the authors discussed their use of a single shared weight across all the connections in the network.

This paper examines the power of network architecture in isolation, without any contribution from synaptic weight training, to solve ML tasks. Specifically, the paper examines the extent to which neural networks with random weights can perform tasks if the architecture has been optimized appropriately. The authors provide a novel algorithm for conducting this optimization on architectures, and show that they can achieve surprisingly good results with random weights (e.g. This paper demonstrates the potential power of architecture optimization procedures, and provides a method for architecture optimization that may be very useful. The results may also be more broadly interesting with respect to questions of seeking appropriate inductive biases in ML. It will be of significant interest to the NeurIPS community.

Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task. We propose a search method for neural network architectures that can already perform a task without any explicit weight training. To evaluate these networks, we populate the connections with a single shared weight parameter sampled from a uniform random distribution, and measure the expected performance.

Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task. We propose a search method for neural network architectures that can already perform a task without any explicit weight training. To evaluate these networks, we populate the connections with a single shared weight parameter sampled from a uniform random distribution, and measure the expected performance.

Have you ever wondered how most mammals are capable of fairly complex tasks, like walking, straight after being born? They haven't had time to experience the world yet so they've clearly not learnt how to perform the actions. Their brains must be pre-wired to enable them to walk, but if the brain structure relies on specific weights then an individual learning from its experiences could lose the ability to act shortly after birth, or never have the ability to begin with. Inspired by this, Adam Gaier and David Ha introduced the world to Weight Agnostic Neural Networks (WANN), an evolutionary strategy for developing neural networks which can perform a task independent of the weights of the connections. In this post, we'll briefly look into Weight Agnostic Neural Networks and use a code implementation to train our very own WANNs on the Lunar Lander gym environment.

In this work we introduced a method to search for simple neural network architectures with strong inductive biases for performing a given task. Since the networks are optimized to perform well using a single weight parameter over a range of values, this single parameter can easily be tuned to increase performance. Individual weight values can then be further tuned as offsets from the best shared weight. The ability to quickly fine-tune weights is useful in few-shot learning and may find uses in continual lifelong learning where agents continually acquire, fine-tune, and transfer skills throughout their lifespan . Early works connected the evolution of weight tolerant networks to the Baldwin effect .